<html>
<head><meta charset="utf-8"><title>tracking performance regressions · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html">tracking performance regressions</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="198350490"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198350490" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198350490">(May 21 2020 at 17:26)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> <span class="user-mention" data-user-id="116083">@pnkfelix</span> Pong. I would suggest that whether we continue to track a regression depends on how severe we consider it. "We generate one extra instruction" isn't necessarily a bug worth tracking. "We fail to vectorize something we vectorized before", on the other hand, is severe enough to be worth tracking.</p>



<a name="198350609"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198350609" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198350609">(May 21 2020 at 17:27)</a>:</h4>
<p>In this case, I don't think the code size issue alone is critical, but generating panic handlers when we didn't before seems problematic.</p>



<a name="198350739"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198350739" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198350739">(May 21 2020 at 17:28)</a>:</h4>
<p>And I suggested reporting it because I wondered if fixing it might be as simple as an <code>#[inline]</code> hint in the right place (and then appropriate testing to make sure that doesn't cause unexpected problems / perf issues).</p>



<a name="198355563"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198355563" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Notification Bot <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198355563">(May 21 2020 at 18:07)</a>:</h4>
<p>This topic was moved here from <a class="stream-topic" data-stream-id="238009" href="/#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bweekly.20meeting.5D.202020-05-21.20.2354818">#t-compiler/meetings &gt; [weekly meeting] 2020-05-21 #54818</a> by <span class="user-mention silent" data-user-id="116122">simulacrum</span></p>



<a name="198355608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198355608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198355608">(May 21 2020 at 18:08)</a>:</h4>
<p>pulled this over here</p>



<a name="198355801"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198355801" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198355801">(May 21 2020 at 18:09)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> One of the problems we have is that "doesn't cause unexpected problems" isn't actually something we can really say too well, unfortunately -- we just don't have a good suite of test cases. For example, <code>#[inline]</code> in panic code can be an optimization if it means it vanishes but <em>definitely</em> hurts compile times for all users of that method (or callers?) as we start to codegen it into downstream crates, which can be quite painful.</p>



<a name="198361662"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361662" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361662">(May 21 2020 at 18:59)</a>:</h4>
<p>Sure, there are tradeoffs between compilation performance and runtime performance.</p>



<a name="198361687"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361687" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361687">(May 21 2020 at 18:59)</a>:</h4>
<p>That's the kind of thing I'd <em>hope</em> could be tied to <code>--release</code>.</p>



<a name="198361833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361833">(May 21 2020 at 19:00)</a>:</h4>
<p>I do think there may be room for --release-but-no-really-i-can-wait</p>



<a name="198361854"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361854" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361854">(May 21 2020 at 19:00)</a>:</h4>
<p>But in any case, what I'm thinking of by "doesn't cause unexpected problems" would be a rust-timer run.</p>



<a name="198361876"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361876" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361876">(May 21 2020 at 19:00)</a>:</h4>
<p>yeah, that's what we'd usually do</p>



<a name="198361887"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361887" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361887">(May 21 2020 at 19:01)</a>:</h4>
<p>Adding the <code>#[inline]</code> and running rust-timer seems worthwhile.</p>



<a name="198361927"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361927" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361927">(May 21 2020 at 19:01)</a>:</h4>
<p>We could then evaluate the tradeoff.</p>



<a name="198361932"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361932" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361932">(May 21 2020 at 19:01)</a>:</h4>
<p>I do think it makes sense to try a rust-timer run here and see how it goes</p>



<a name="198361947"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361947" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361947">(May 21 2020 at 19:01)</a>:</h4>
<p>(presuming there's an inline or whatever that can be added)</p>



<a name="198361982"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198361982" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198361982">(May 21 2020 at 19:01)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/131828-t-compiler/topic/tracking.20performance.20regressions/near/198361833">said</a>:</p>
<blockquote>
<p>I do think there may be room for --release-but-no-really-i-can-wait</p>
</blockquote>
<p>Perhaps, yes. Among other things, that could also turn on LTO.</p>



<a name="198362087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198362087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198362087">(May 21 2020 at 19:02)</a>:</h4>
<p>We'd probably want to start by adding a new optimization level to rustc. But I don't know how much this is on-topic for <em>this</em> particular issue.</p>



<a name="198362331"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198362331" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198362331">(May 21 2020 at 19:04)</a>:</h4>
<p>Yes for this particular case I would agree that trying to fix it seems reasonable</p>



<a name="198362359"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198362359" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198362359">(May 21 2020 at 19:04)</a>:</h4>
<p>but I think if there's not an easy fix (e.g. inline annotations) then there's not much to do</p>



<a name="198362432"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198362432" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198362432">(May 21 2020 at 19:05)</a>:</h4>
<p>I do <em>absolutely</em> agree that we can't track every last "why does new rustc generate three more instructions here" regression, and we should focus on the ones that have an impact.</p>



<a name="198362459"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198362459" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198362459">(May 21 2020 at 19:05)</a>:</h4>
<p>There's definitely a (fuzzy) line somewhere for what we should track.</p>



<a name="198362770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198362770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198362770">(May 21 2020 at 19:08)</a>:</h4>
<p>On a different note...I wonder how difficult it would be to have a fully automated regression-bisect bot? For instance, "here's a test case, it's a failure if the compiled output {includes this instruction, doesn't include this instruction, calls this function, has more than X instructions, ...}". @ some github bot, get back a commit hash and a diff of compiled output for the test case.</p>



<a name="198363129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198363129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198363129">(May 21 2020 at 19:11)</a>:</h4>
<p>Once you add support to <code>cargo bisect-rustc</code>, adding support to my bisect bot should just be a matter of parsing the instruction for it and passing it to <code>cargo bisect-rustc</code>.</p>



<a name="198363801"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198363801" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198363801">(May 21 2020 at 19:16)</a>:</h4>
<p>do we expect a lot of these regressions to be in rustc rather than the one PR updating LLVM to the most recent version ?</p>



<a name="198363878"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198363878" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198363878">(May 21 2020 at 19:17)</a>:</h4>
<p>it's mostly std or llvm, these days, but as we have more mir opts that'll probably change</p>



<a name="198363894"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198363894" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198363894">(May 21 2020 at 19:17)</a>:</h4>
<p>right</p>



<a name="198364019"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198364019" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198364019">(May 21 2020 at 19:18)</a>:</h4>
<p>codegen changes in std happen more often than the others, very true</p>



<a name="198364223"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198364223" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198364223">(May 21 2020 at 19:20)</a>:</h4>
<p>I guess lolbench could also have a role to play in this conversation</p>



<a name="198364762"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198364762" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198364762">(May 21 2020 at 19:25)</a>:</h4>
<p>(although its regressions had a per-nightly granularity IIRC rather than per-commit; and here the actual code regressing would be known to the bisector, rather than some performance counters stats)</p>



<a name="198365532"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198365532" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198365532">(May 21 2020 at 19:31)</a>:</h4>
<p>lolbench iirc takes forever to run</p>



<a name="198365535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198365535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198365535">(May 21 2020 at 19:31)</a>:</h4>
<p>like, days</p>



<a name="198365572"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198365572" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198365572">(May 21 2020 at 19:31)</a>:</h4>
<p>yeah, part of the reason why it couldn't per-commit iirc</p>



<a name="198365728"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198365728" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198365728">(May 21 2020 at 19:33)</a>:</h4>
<p>(maybe with enough sponsored hardware + pinning benches to the same host it could be speedier :)</p>



<a name="198366138"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/tracking%20performance%20regressions/near/198366138" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/tracking.20performance.20regressions.html#198366138">(May 21 2020 at 19:36)</a>:</h4>
<p>perhaps :)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>